Security, How not to react....

I'm not sure Paul has done himself many favours in his reactions this week. Security breaches happen all the time, either from relatively friendly testing, or more frequently actual breakins. He should be breathing a sigh of relief that the worst damage is having to run a SQL statement DELETE * FROM xxx where body like "%test_xss%"

Last year I blogged about one of my clients getting rootkit'ed, this is probably the worst of all Unix based disasters. And the lessons from that have to some degree been applied and acted on.

• Keep up to date with security announcements (and schedule updates reasonably regularly
• run chkrootkit on a cronjob (as debian's package does now)
• Keep a good set of backups
• Try keeping up a backup server (with replicates the main servers operations)
• Practice taking the backup server live!

This year's issues saw yet more attacks, most common has been ssh port scanning and user testing. Looking at /var/log/auth.log, a few months ago I started noticing a huge number of failures on the main web server. And not just from one IP address. As a defensive measure I did one simple thing

• Change the port that ssh runs on!

While this works well on my server, one of my main clients has also implemened ip based restrictions at the firewall level for ssh. So unless you are a registered IP address, you do not even get a prompt.

I had made this modification to a couple of boxes that I directly use, however It was not feasible for a number of others that i rarely touch.

Last month, I saw the result of what the ssh attackers had started doing. From the logs a Romanian IP had managed to log in a few times under the ftp account via ssh (An account that was probably briefly enabled when I set up ftp, but latter ended up using .htaccess style passwords for it), but had forgotton about. Obviously the password I had given that account had been less than secure, and they had managed to get a shell account.

Thanks to the wonderfull logging ability of Linux/Unix etc., I had a facinating history of what they had attempted (via the bash_history) and copies of quite a few of the files they used. (funnily enough they had to use the ftp home directory, which was a highly visable location to put .tgz files on a windows network!)

I guess partly by luck they had not attempted to rootkit the box, although the system was quite up-todate (tested via chkrootkit, and looking at the logs). They had however tried installing an irc server, and used the box as a scanner to find other vunerable ssh hosts. (the irc didnt work due to the user privages mixed with our firewall configuration.) I didnt examine the logs that much to see if they found any other vunerable hosts.

So what lessons came out of this one, - another run around check on the servers I look after

• start using the AllowUsers line in the ssh file.
• where feasible lock down ssh access to IP's at the firewall level
  
• regulary check /etc/passwords to see who has accounts.. and shells.
• regulary check /var/log/auth.log with grep to see who has been accessing (or trying to access the server)

In the same respects while the wiki at pear was being tested, I was on IRC with they people doing the testing, and was babling on how half the problem with pear and the wiki was that it used PHP as templates, (while not totally true, it is a big enough factor to make it a concern).

The real problem I had spotted months ago when fixing a report of another XSS attack on the pear site, was the disjoin between input and output. Variables where being set in one place, and used in another file, with no basic premise if they had been escaped or not. Something I had been ranting about, with only a few takers, when describing why template engines where valuable. The basic premise that all data should be untrusted and hence escaped, unless you expressly know it's ok... (rather than the other way round..!)

To say I'm a guru on this however would be twisting words, during the tests that where being run on pear, the testers, fired it off at my site, and one of my clients (at my request). The hole they found on my site was due to a error message not escaping input correctly (something I would not normally do, but had hacked up my own site a bit too hastely), however, it appears that most other attacts failed due to flexy's escaping, and maybe in part to the javascript defence on the comment system.

My client faired a little worse, as we had been using a very old version of the Pager library, which has since been updated, along with some rather silly mistakes that had been made with the url writing being done in PHP rather than the template engine.

But this is definatly not an invitation to try and hack my systems, but if you do, please dont rootkit them, as it's a royal Pain in the ass to fix!!!